PMM-15331 Gate health on SEP provisioning - #5768

With PMM_ENABLE_SEP set, pmm-server now provisions the Grafana service
account SEP authenticates users against and publishes its token to
/srv/sep-secrets, one file per canonical SEP settings name. This replaces
the manual minting step SEP's feature-build harness performs today with
admin:admin basic auth from outside the container, which breaks as soon as
the Grafana admin password changes.

The account is created directly in Grafana's own PostgreSQL database over
the local socket, the same way postgres-sep provisions the sep role as the
local superuser: Grafana exposes no credential-free path to create a
service account, and pmm-server holds no Grafana admin credential once an
operator has changed the admin password.

The helper runs as a supervisord one-shot rather than from the entrypoint,
because Grafana is neither running nor migrated - on a first boot its
schema does not exist at all - until long after the entrypoint has exec'd
supervisord. The token is revalidated against Grafana on every start and
replaced only when Grafana rejects it, since SEP reads the secrets
directory once at its own start and re-minting unconditionally would
revoke the token a running SEP is still holding.

A freshly minted token that Grafana rejects is fatal and is never
published, so a future change to Grafana's schema or token encoding
surfaces as a failed sep-provision instead of silent SEP auth failure.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Stop a failed provisioning run from respawning without bound. startsecs = 0
is required for a one-shot, but it also means the program never enters
BACKOFF, which is the only state startretries bounds - so autorestart =
unexpected respawned every non-zero exit immediately and forever, measured
at ~10 runs per second. That turned the deliberate exit-1 safety net into a
hot loop of pbkdf2 and api_key writes against Grafana's database. A failed
run is meant to be read, not retried, so autorestart is now false.

Re-assert the service account's org role on the keep-existing-token path.
The validator endpoint answers 200 for any role, so an account demoted in
the Grafana UI kept authenticating while SEP's org-role lookups silently
degraded.

Make the secrets directory group-writable (0770). The image supports an
arbitrary runtime uid in group 0, and 0750 left such a uid unable to write
the mountpoint shipped in the image, which is also why no supervisord
program in this repo pins user = pmm. Matches the permissive mode the other
/srv directories already use.

Also bound the authentication call with --max-time, remove the temporary
file if publishing fails part-way, keep the disabled path inert when the
directory is not writable, and correct the documented re-mint semantics:
the token is replaced when the file is missing or empty too, not only when
Grafana rejects it.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Report, rather than swallow, a token file that cannot be removed when
PMM_ENABLE_SEP is unset. The disabled path must not fail a start over a
directory this uid cannot write, but a token left in place is still
readable by SEP, so silence there contradicted the documented contract.

Raise the Grafana readiness budget to 600s. A one-shot cannot bound its own
restarts under supervisord, so a run that gives up is not retried until the
container restarts; waiting longer is what absorbs a slow first boot, which
is the transient failure this is actually exposed to.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Puts SEP's two remaining deployment secrets through the secrets directory
PMM-15280 established, one file per canonical SEP settings name, so
neither travels through the SEP container's environment where docker
inspect and every process in its PID namespace can read them.

The lifecycles differ deliberately. SECRET_KEY is generated once and
persisted on /srv, following /srv/.postgres_password: re-minting it signs
out every SEP session and changes the inter-service token derived from it
by HMAC. The three *__DATABASE__PASSWORD files are rewritten on every
start so rotating PMM_SEP_POSTGRES_PASSWORD takes effect on the next
restart.

The helper runs from the entrypoint rather than under supervisord because
neither value needs a running service, which puts both on disk before
supervisord starts and keeps them clear of the Grafana readiness wait.
Unlike grafana-sep it does not bail on an external Grafana database:
postgres-sep still provisions the sep role there, so SEP still needs its
password files.

CELERY__BEAT_DBURI is not written. SEP-1797 landed, so the beat store
resolves from the SEP database settings and percent-encodes the password
itself; an explicit URI here would persist as an override.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Five defects found reviewing the helper against the permission and
re-mint contracts it documents.

The published files took the group of whichever uid pmm-server runs
as, not the group of the directory, so the documented group_add: ["0"]
requirement only held while that uid's primary group was 0 - which is
true of the stock image and false of the arbitrary-uid deployments the
entrypoint supports. The mountpoint is now setgid, which Docker
carries into a fresh named volume, so the files belong to group 0
whatever uid writes them.

The reuse check read an unreachable Grafana as a rejected token and
re-minted, revoking the token a running SEP holds - the outcome the
design exists to prevent. The probe now reports the status it got,
retries a call it could not make, and treats an inconclusive answer as
fatal rather than as a rejection.

install -d on an existing directory also chmods it, which a uid that
does not own the mountpoint cannot do; it exited 0 only because the
mode it asked for happened to match the image. Creation and the
writability check are now separate, so the fatal fires only when the
directory really cannot be used.

On the keep path a psql failure in the secondary role repair aborted
under errexit before the already-valid token was republished; the two
are now in dependency order. The Grafana-not-on-embedded-Postgres
guard now removes the token files like the disabled path does, rather
than leaving a token it can no longer revalidate.

Also renames the directory override to PMM_SEP_SECRETS_DIR, which no
longer collides with the name SEP uses for its own mount, and records
in .env.example that the settings names must be left unset rather than
empty on the SEP side or the mounted files go unused.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Narrowing the persisted key back to 0600 was unconditional and therefore
errexit-fatal. chmod needs ownership, not write permission, and the
arbitrary-uid path lets a later start run as a uid that does not own what
an earlier one persisted - so a key this start could read and publish
perfectly well stopped pmm-server from booting instead. Verified against
percona/pmm-server: a 0640 key owned by uid 1000, read as uid 1002,
exited 1 and published nothing. entrypoint.sh guards its own /srv chmod
the same way this now does, and postgres-migration only chmods a
credential file it just created.

Softening that alone would have opened a second hole, since a command
substitution failing in argument position does not trip errexit: an
unreadable key would have published an empty SECRET_KEY over a good one.
The value is now read into a variable and checked before publishing.

Also documents in .env.example what an operator otherwise finds out the
hard way: where the key is persisted, that whitespace around
PMM_SEP_POSTGRES_PASSWORD is kept by PostgreSQL but stripped by SEP, and
that with SEP enabled these failures stop PMM Server rather than
degrading SEP alone - which the secrets directory becoming an entrypoint
concern is what changed.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Cuts the review fixes' comment additions from 16 lines to 10: drops
the remove_token_files note, whose symmetry with the disabled branch
six lines above is visible; drops the aside about setups that mount
nothing at the secrets path; and folds three restatements of the
group-0 requirement into one.

What is left names the facts a reader would otherwise undo: that the
setgid bit is what makes the group inheritance work, that unreachable
and rejected must stay distinct, that install -d chmods a directory it
did not create, the errexit ordering on the keep path, and that the
empty case arm is intentional.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Drop PMM_SEP_POSTGRES_PASSWORD from the environment once the last
consumer has run. It otherwise reaches supervisord and every child,
and pmm-managed-init trace-logs each variable it is handed, name and
value included, so PMM_TRACE would print the SEP database password.
Mirrors the POSTGRES_PASSWORD unset earlier in the entrypoint.

Also correct the .env.example wording for the two fatal conditions:
say plainly that an unwritable /srv/sep-secrets stops PMM Server, and
record that neither check runs under PMM_HA_ENABLE or
PMM_DISABLE_BUILTIN_POSTGRES, where PMM_ENABLE_SEP is ignored outright.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Drops the header paragraph restating the revalidation rule the
grafana_auth_status docblock already carries, and the two lines
restating a SQL predicate and an idempotency guard visible in the
statements below them. Tightens the rest.

Also corrects the DELETE comment, which described only the arm that
avoids the name collision and not the one that removes every token on
the account.

47 comment lines against 187 of code, from 63. Still above the 0.15 of
postgres-sep, which the remaining blocks account for: this script
writes another product's schema and reproduces its token encoding, so
the index names, the psql interpolation rule, the api_key.role
behaviour and the reason for touching the database at all are facts no
code in this repo can show.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

publish() interleaved the write and the rename per file, so a write that
died part-way through the set - a full volume being the realistic cause -
left SEP some files rotated and the rest stale. Verified against the
previous revision: a failure on the third password file rotated two of
them and left the third behind.

Split it into stage() and commit_staged(). Every fallible step now runs
before the first rename, so the same failure leaves the previous complete
set in place and the script still exits nonzero. The temporary is recorded
for cleanup immediately after mktemp rather than after the write, since
errexit abandons the rest of stage() on a failed write and an unrecorded
temporary is one the EXIT trap cannot find.

This narrows rather than closes the gap: the set is still four renames, so
a failure between them leaves the earlier files updated. Closing that needs
a directory swap the mountpoint contract rules out.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

The sep-provision supervisord unit this branch adds starts unconditionally
and is a no-op when PMM_ENABLE_SEP is unset, so its stdout log always
exists under /srv/logs and always reaches the archive. TestDownloadLogs
compares the archive against an exact list, so it fails on every build
until the file is named there.

Caught by pmm3-api-tests #7005 on the feature build.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

The container's HEALTHCHECK moves from an inline curl to
/opt/healthcheck.sh, which runs the same readyz probe and, only when
PMM_ENABLE_SEP is set, additionally requires a marker proving this
start's provisioning run reached a successful conclusion.

grafana-sep writes /srv/.sep_provisioned from its EXIT trap, keyed on
its own exit status, so every deliberate exit 0 publishes it and every
FATAL path leaves none. entrypoint.sh removes the marker before
exec supervisord, so its presence can only describe the current start.

The one exception is the guard that skips provisioning on a non-embedded
Grafana database, where the marker decision mirrors sep-secrets' own
precedence. Under PMM_HA_ENABLE or PMM_DISABLE_BUILTIN_POSTGRES the
marker is written and the container still reports healthy, whatever
GF_DATABASE_* says: sep-secrets bails on those two first and removes
every file SEP has, so SEP refuses to start rather than running inert,
and the documented external-PostgreSQL recipe sets both. Under
GF_DATABASE_URL or GF_DATABASE_HOST alone the marker is withheld, since
sep-secrets still publishes SEP's four password files there and a
released side-car would come up with Grafana authentication inert.

--start-period widens from 25s to 720s to cover grafana-sep's nominal
600s Grafana wait plus its bounded token validation. This delays only
the unhealthy verdict for a server that has never been ready; a probe
that succeeds during the start period marks the container healthy
immediately, as it does today.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

The fatal fired on PMM_ENABLE_SEP alone, so an unremovable marker took the
container down under PMM_HA_ENABLE or PMM_DISABLE_BUILTIN_POSTGRES - the two
configurations the lines just above declare PMM_ENABLE_SEP ignored in, and
where grafana-sep marks the run whatever provisioning did. A marker it cannot
remove there misleads nothing, and refusing to start is worse than the outage
the gate is meant not to cause.

An external Grafana database alone stays fatal: grafana-sep withholds the
marker for it, so a stale one would hold the container green for good.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

PMM_ENABLE_SEP is ignored outright under PMM_HA_ENABLE or
PMM_DISABLE_BUILTIN_POSTGRES, so the health gate now tests that whole
predicate rather than the bare flag. It used to test the flag alone and rely
on grafana-sep marking a run it had skipped to release a gate that should
never have applied - the criterion held, but only because the skip branch
happens to exit within seconds.

Both skip paths therefore stop marking the run, including the long-standing
one for PMM_ENABLE_SEP unset, which marked a start whose token files it had
just deleted. The marker now means what it says: this start provisioned.

An external Grafana database alone is deliberately not exempt. SEP does start
there, with its Grafana authentication inert, which is the case the gate
exists to catch.

.env.example gains the configuration it omitted - an external Grafana
database never reports healthy - and drops the claim that health is
unchanged under the two ignore flags, which was true only after
sep-provision had run. The 720s start period is now stated as applying to
every deployment.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

The two preceding commits put back eight of the twelve comment lines the trim
had removed, in changes that were not about comments. Removed again, below the
post-trim baseline.

The marker-clear comment needed no edit at all: "Fatal only when SEP is in
use" was inaccurate before the condition was narrowed and accurate after, so
narrowing the condition was the whole fix. The healthcheck's new block
restated its own condition and repeated a warning entrypoint.sh already
prints. The cleanup comment keeps only the $?-capture ordering, which is the
one part not inferable from the code; MARK_RUN says what it does at each site
that clears it.

.env.example keeps the case it was missing - an external Grafana database
never reports healthy - and drops the start-period explanation, which is
image behaviour rather than something an operator sets here.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>

Where READY_TIMEOUT is set rather than in the Dockerfile: the reader who
raises it is the one who needs to know the image's start-period follows it.

Signed-off-by: Yan Orestes <yan.orestes@percona.com>